Graph Representation Querying of Machine Learning Models for Traffic or Safety Rules

ABSTRACT

A graph representation of a tactical map representing a plurality of static components of an environment of a vehicle is generated. Nodes of the graph represent static components, and edges represent relationships between multiple static components. Different edge types are used to indicate respective relationship semantics among the static components. Individual nodes are represented as having the same number and types of edges in the graph. Using the graph as input to a neural network based model, a set of results is obtained. A motion control directive based at least in part on the results is transmitted to a motion-control subsystem of the vehicle.

This application is a continuation of U.S. patent application Ser. No.16/143,117, filed Sep. 26, 2018, which claims benefit of priority toU.S. Provisional Application No. 62/564,160 filed Sep. 27, 2017, whichare hereby incorporated by reference in their entirety.

BACKGROUND Technical Field

This disclosure relates generally to systems and algorithms forreasoning, decision making and motion planning with respect tocontrolling the motion of autonomous or partially autonomous vehicles.

Description of the Related Art

Motorized vehicles which are capable of sensing their environment andnavigating to destinations with little or no ongoing input fromoccupants, and may therefore be referred to as “autonomous” or“self-driving” vehicles, are an increasing focus of research anddevelopment. Until relatively recently, due to the limitations of theavailable hardware and software, the maximum speed at which computationsfor analyzing relevant aspects of the vehicle's external environmentcould be performed was insufficient to enable non-trivial navigationdecisions to be made without human guidance. Even with today's fastprocessors, large memories, and advanced algorithms, however, the taskof making timely and reasonable decisions (which are based neither onexcessively pessimistic assumptions, nor on excessively optimisticassumptions) regarding an autonomous vehicle's trajectory in the contextof unpredictable behaviors of other entities (such as other drivers orother autonomous vehicles) and incomplete or noisy data about static anddynamic components of the vehicle's environment remains a significantchallenge.

SUMMARY OF EMBODIMENTS

Various embodiments of methods and apparatus for analyzing graphrepresentations of tactical maps for autonomous vehicles using neuralnetwork-based machine learning models are described. According to someembodiments, a tactical map may comprise information about variousstatic components of a vehicle's operating environment, such as roadlane segments, intersections, and so on. The information included in atactical map may indicate attributes or properties of individual staticcomponents as well as various types of relationships (e.g., geometric ortopological relationships) that may exist among the static components.From the raw tactical map, a homogenized graph representation may begenerated in various embodiments, suitable for processing by a neuralnetwork model which has been trained to perform reasoning on graphs orgraph-like data structures. In the homogenized graph, nodes mayrepresent instances of the static components, and edges may representrelationships (with respective edge types denoting relationships withdifferent semantics). The graph may be considered to be homogenized invarious embodiments in that individual nodes may be represented ashaving the same number of edges with the same edge types as other nodes,arranged in the same order, within the graph representation. In effect,in such embodiments, missing edges among nodes may be represented usingconnections from such nodes to a special “zero” node. Suchhomogenization may help to simplify some of the computations performedat the neural network (e.g., by reducing the number of distinctparameters which have to be learned) in some embodiments. The results ofthe analysis of the graph corresponding to a tactical map may, forexample, be combined with results of analyses of other environmentcomponents (such as moving vehicles in the vicinity, pedestrians and thelike) to make decisions regarding possible or advisable future motionsof the vehicle in some embodiments. Such decisions may be implemented bysending the appropriate motion control directives to varioussubcomponents of the vehicle (such as braking subsystems, turningsubsystems, accelerating subsystems, and the like) in such embodiments.

According to at least one embodiment, a method may comprise obtaining afirst tactical map associated with an environment of a first vehicle.The first tactical map may indicate at least a plurality of staticcomponents of the environment. The method may comprise generating ahomogenized graph representation of the first tactical map comprising aplurality of nodes and a plurality of edges. Individual ones of thenodes may represent respective static components, and individual ones ofthe edges may represent respective relationships between multiple staticcomponents (e.g., pairs of static components). An edge type indicativeof relationship semantics of a first edge associated with a first nodeof the graph may differ from an edge type of a second edge associatedwith the first node. The graph may be constructed such that the numberof edges associated with the first node of the graph is equal to thenumber of edges associated with individual ones of one or more othernodes of the graph. The method may comprise obtaining, using thehomogenized graph representation as input to a neural network-basedmachine learning model, at least a first set of reasoning resultspertaining to the first vehicle and one or more static components. Themethod may also include transmitting, to a motion-control subsystem ofthe first vehicle, one or more motion-control directives based at leastin part on the first set of reasoning results. Over time, in someembodiments, as the vehicle moves during a given journey, additionaltactical maps relevant to the current position of the vehicle may beobtained and analyzed analogously using the neural network-based machinelearning model.

In at least some embodiments, the neural network-based machine learningmodel may comprise one or more convolution layers. In one embodiment, apropagation function of a particular convolution layer may comprise anon-linear function applied to a linear combination of nodes with theirneighbors (with the various nodes being represented by respectivefeatures). In one embodiment, computations of a convolution layer maycomprise a summation, over one or more edge types, of a product of atleast (a) an adjacency matrix associated with an edge type (b) inputreceived from another layer of the neural network (which, in the case ofa first layer of the network, may comprise a feature matrix representingproperties of graph nodes) and (c) a vector representing learned weightsassociated with respective edge types. Other types of neural networksconfigured to process graph data structures, such as relational neuralnetworks, and/or non-convolutional graph neural networks may be used insome embodiments, e.g., instead of or in addition to neural networkswith convolution layers.

According to one embodiment, a system may comprise one or more computingdevices. The computing devices may be configured to generate ahomogenized graph representation of a tactical map. The tactical map maycomprise information pertaining to a plurality of static components ofan environment of a vehicle. The homogenized graph representation maycomprise a plurality of nodes and a plurality of edges, with individualones of the nodes representing respective static components, andindividual ones of the edges representing respective relationshipsbetween multiple static components. An edge type indicative ofrelationship semantics of a first edge associated with a first node maydiffer from an edge type of a second edge associated with the firstnode. The computing devices may also be configured to obtain, using atleast a portion of the homogenized graph representation as input, a setof results from a neural network-based machine learning model. One ormore motion-control directives based at least in part on the set ofresults may be transmitted to a motion-control subsystem of the vehicle.

According to some embodiments, a non-transitory computer-accessiblestorage medium may store program instructions that when executed on oneor more processors cause the one or more processors to generate ahomogenized graph representation of a tactical map. The tactical map maycomprise information pertaining to a plurality of static components ofan environment of a vehicle. The homogenized graph representation maycomprise a plurality of nodes and a plurality of edges, with individualones of the nodes representing respective static components, andindividual ones of the edges representing respective relationshipsbetween multiple static components. An edge type indicative ofrelationship semantics of a first edge associated with a first node maydiffer from an edge type of a second edge associated with the firstnode. The instructions when executed on the processors may further causethe processors to obtain, using at least a portion of the homogenizedgraph representation as input, a set of results from a neuralnetwork-based machine learning model. One or more motion-controldirectives based at least in part on the set of results may betransmitted to a motion-control subsystem of the vehicle.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 illustrates an example system environment in which graphrepresentations of tactical maps representing static components of anautonomous vehicle's environment may be analyzed using neuralnetwork-based models to help direct the movements of the vehicle,according to at least some embodiments.

FIG. 2 illustrates an example decision making scenario for an autonomousvehicle, as well as general problem characteristics associated with suchdecision making scenarios, according to at least some embodiments.

FIG. 3 illustrates an example overview of the processing of tacticalmaps at an autonomous vehicle, according to at least some embodiments.

FIG. 4 illustrates example categories of static components of theenvironment which may be included in a tactical map, as well as examplesof properties and relationships of one of the categories, according toat least some embodiments.

FIG. 5 illustrates an example neural network architecture which may beused to process homogenized graph representations of tactical maps,according to at least some embodiments.

FIG. 6 illustrates a simple example of a feature matrix which may begenerated to represent nodes of a homogenized graph corresponding to atactical map, according to at least some embodiments.

FIG. 7 illustrates a set of adjacency matrices which may be generated torepresent edges of the homogenized graph corresponding to the tacticalmap introduced in FIG. 6 , according to at least some embodiments.

FIG. 8 illustrates an intermediate summation operation associated with apropagation function of a first convolution layer at a neuralnetwork-based machine learning model to which the homogenized graphdiscussed in the context of FIG. 6 and FIG. 7 is provided as input,according to at least some embodiments.

FIG. 9 illustrates example queries to which responses may be generatedusing a neural network-based machine learning model similar to thatdiscussed in the content of FIG. 8 , according to at least someembodiments.

FIG. 10 illustrates an overview of example stages of developing,deploying and using machine learning models for autonomous vehicles,according to at least some embodiments.

FIG. 11 illustrates aspects of operations which may be performed toanalyze graph representations of tactical maps at autonomous vehiclesusing neural network-based machine learning models, according to atleast some embodiments.

FIG. 12 is a block diagram illustrating an example computing device thatmay be used in at least some embodiments.

While embodiments are described herein by way of example for severalembodiments and illustrative drawings, those skilled in the art willrecognize that embodiments are not limited to the embodiments ordrawings described. It should be understood, that the drawings anddetailed description thereto are not intended to limit embodiments tothe particular form disclosed, but on the contrary, the intention is tocover all modifications, equivalents and alternatives falling within thespirit and scope as defined by the appended claims. The headings usedherein are for organizational purposes only and are not meant to be usedto limit the scope of the description or the claims. As used throughoutthis application, the word “may” is used in a permissive sense (i.e.,meaning having the potential to), rather than the mandatory sense (i.e.,meaning must). Similarly, the words “include,” “including,” and“includes” mean including, but not limited to. When used in the claims,the term “or” is used as an inclusive or and not as an exclusive or. Forexample, the phrase “at least one of x, y, or z” means any one of x, y,and z, as well as any combination thereof.

DETAILED DESCRIPTION

FIG. 1 illustrates an example system environment in which graphrepresentations of tactical maps representing static components of anautonomous vehicle's environment may be analyzed using neuralnetwork-based models to help direct the movements of the vehicle,according to at least some embodiments. As shown, system 100 comprisesan autonomous or partially-autonomous vehicle 110. The term “autonomousvehicle” may be used broadly herein to refer to vehicles for which atleast some motion-related decisions (e.g., whether to accelerate, slowdown, change lanes, etc.) may be made, at least at some points in time,without direct input from the vehicle's occupants. In variousembodiments, it may be possible for an occupant to override thedecisions made by the vehicle's decision making components, or evendisable the vehicle's decision making components at least temporarily;furthermore, in at least one embodiment, a decision-making component ofthe vehicle may request or require an occupant to participate in makingsome decisions under certain conditions. Vehicle 110 may compriseseveral categories of components in the embodiment depicted in FIG. 1 :motion-related decision making components 116, local sensor collection112, perception subsystem 113, motion control subsystems 120 andcommunication devices 114. The decision making components 116 maycomprise multiple tiers or layers in the depicted embodiment, includingbut not necessarily limited to a behavior planner 117, a motion selector118 and a state prediction subsystem 133. In the depicted embodiment,one or more of the decision making components 116 may comprise orutilize machine learning models which are generated or trained at leastin part using data center-based machine learning resources 175. Asshown, state prediction subsystem 133 may include a plurality of machinelearning models, including tactical map analysis model(s) 135 anddynamic entity models 136 in some embodiments.

As discussed below in further detail, an autonomous vehicle 110 mayobtain and/or store one or more tactical maps which comprise informationabout static components of the vehicle's operating environment(including, for example, properties or attributes of the staticcomponents and relationships among the static components) in variousembodiments. The static components may include, for example,road/highway lane segments, intersections, traffic signs, and the like.During the course of a journey from some location L1 to some destinationlocation L2, a sequence of tactical maps may be obtained or analyzed atthe autonomous vehicle using models 135, with each of the tactical mapscorresponding to changed intermediate locations of the vehicle 110.

Corresponding to a given tactical map, in various embodiments ahomogenized graph representation may be created, in which nodesrepresent static components of the map, and edges representrelationships. Sub-groups of edges of the graph may have respective edgetypes indicative of the different semantics of the relationship beingrepresented—e.g., one group of edges may represent the relationship“is-to-the-left-of” between a pair of lane segments LS1 and LS2, anothergroup of edges may represent the relationship “is-to-the-right-of”, andso on. The graph may be considered homogenized in that, at least in someembodiments, individual ones of the nodes may have the same number andtypes of edges associated with them. The node for each static componentmay be assumed to have a certain set of edges to respective destinationnodes representing other static components, and if a static componentcorresponding to the destination of a particular edge is not present inthe input tactical map, then that destination may be set to a specialzero node in various embodiments. An edge connecting a given pair ofnodes may represent geometric or topological constraints associated withthe nodes in various embodiments (e.g., constraints on the distancesbetween the objects representing the nodes, the directions in which theobjects are oriented with respect to each other, the heights orelevations of the objects with respect to one another, etc.) Attributesmay be associated with nodes as well as with edges—for example, nodeattributes may include speed limits, while edge attributes may includelane permeability, or rules regarding giving way when changing lanes.The manner in which information about nodes and edges may be encodednumerically in some embodiments is explained in further detail belowwith the help of a simple example.

The homogenized graph representation of a given tactical map may beprovided as input to a neural network-based machine learning model 135in various embodiments. The model 135 may, for example, have beentrained using data center-based training resources 175 in someembodiments to generate predictions (or scores) for responses to varioustypes of queries pertaining to the information contained in a tacticalmap, such as queries regarding the feasibility of transitions from onelane to another, the paths that can be taken (in terms of consecutivelytraversed lane segments) to attain some short-term objective, and so on.In at least some embodiments, the set of results generated using themodel 135 may be used (e.g., in combination with analysis pertaining todynamic components of the environment such as other moving vehicles,pedestrians etc., as well as other computations performed at othermotion-related decision components 116) to determine the motion-controldirectives 134 which cause the vehicle to move along a particulartrajectory. In various embodiments, the analysis of staticinfrastructure elements indicated in a tactical map may form a subset ofthe overall computations needed to help decide upon and achieve thevehicle trajectories.

In the depicted embodiment, the motion control subsystems 120, such as abraking system, acceleration system, turn controllers and the like maycollectively be responsible for causing various types of movementchanges (or maintaining the current trajectory) of vehicle 110, e.g., inresponse to directives or commands issued by the motion selector 118and/or the behavior planner 117 using state predictions obtained fromsubsystem 133, resulting in the actual or realized trajectory 122. Inthe tiered approach towards decision making illustrated in FIG. 1 , themotion selector 118 may be responsible for issuing relativelyfine-grained motion control directives 134 to various motion controlsubsystems. The rate at which directives are issued to the motioncontrol subsystems 120 may vary in different embodiments—for example, insome implementations the motion selector 118 may issue one or moredirectives approximately every 100 milliseconds, which corresponds to anoperating frequency of about 10 Hertz for the motion selector 118. Ofcourse, under some driving conditions (e.g., when a cruise controlfeature of the vehicle is in use on a straight highway with minimaltraffic) directives to change the trajectory may not have to be providedto the motion control subsystems at some points in time. For example, ifa decision to maintain the current velocity of the vehicle is reached bythe decision making components based on state predictions from models133, and no new directives are needed to maintain the current velocity,the motion selector may not issue new directives every T millisecondseven though it may be capable of providing such directives at that rate.

The motion selector 118 may determine the content of the directives tobe provided to the motion control subsystems (i.e., whether braking toslow speed by X units is required, whether acceleration by Y units isrequired, whether a turn or lane change is to be implemented, etc.)based on several inputs in the depicted embodiment, includingconditional action and state sequences generated by the behavior planner117 (as indicated by arrow 119), data obtained from sensor collection112 via perception subsystem 113, and predictions of future states ofthe environment of the vehicle 110 generated by subsystem 133. The term“world state” may also be used to refer to the domain over whichpredictions are generated by subsystem 133 in various embodiments. In atleast some embodiments, the world state may include the tactical map,representations of the states of various other moving entities (e.g.,nearby vehicles, some of which may also be autonomous orsemi-autonomous) as well as stationary objects that could potentiallyimpact the decisions to be made regarding the movements of vehicle 110.In one embodiment, the state prediction subsystem 133 may notnecessarily be bundled with the behavior planner 117 and/or the motionselector 118 as part of a decision-making subsystem; instead, the stateprediction subsystem 133 may be considered another example of an inputsource for the decision-making subsystem.

According to at least some embodiments, the task of generating model(s)to be used at subsystem 133 may involve several phases, some of whichmay be performed continuously or in parallel with others. For example,in various embodiments, data indicating static components of theenvironment at thousands or millions of locations, data pertaining tothe driving behavior of agents controlling vehicles (which may includehuman drivers as well as autonomous driving systems) under a widevariety of circumstances and external driving conditions, and so on, maybe collected over months or years or on an ongoing basis, forming apotentially very large input data set for successive iterations ofmachine learning model improvements. Respective portions of such datamay be in various formats—e.g., in the form of videos or still imagescollected from on-vehicle cameras or stationary cameras outsidevehicles, output from LIDAR (light detection and ranging) devices, radarand other types of local sensors, sensors attached to the vehicle motioncontrol subsystems to capture braking, acceleration and turningpatterns, global positioning devices, location sensors based on wi-fi orother signals, and so on. Data may also be collected from variousdatabases such as law repositories for various jurisdictions in whichautonomous vehicles are expected to operate, map repositories, accidentreport databases, motor vehicle or road management departments ofgovernments, and so on.

In various embodiments, the analysis of the tactical maps using models135 may be combined with the analysis of a varying number of dynamic ormoving entities/components of the environment (e.g., the autonomousvehicle for which state information is being collected, other vehicles,pedestrians, bicyclists, and the like) using models 137. Individual onesof the moving entities may be represented using respective state vectorsin some embodiments. In at least some embodiments, neural network-basedmachine learning models may be used for dynamic entities as well astactical maps (or a single combined neural network model may be trainedfor analyzing both static and dynamic components/entities). Models 135and 137 may collectively be used to generate probabilistic predictionsregarding future world states in the depicted embodiment.

After a model 135 or 137 has been trained and evaluated, e.g., usingdata center based resources 175, it may be deployed for execution at oneor more autonomous vehicles 110 in the depicted embodiment. In someembodiments, a subset of the model's parameters may be left open orun-finalized at the time of deployment to an autonomous vehicle fleet;values for such parameters may be determined at the autonomous vehicleitself. At the vehicle, input collected from local sensors 112 andcommunication devices 114 may be provided to the model(s) 133 (as wellas to other decision making components such as the behavior planner 117and motion selector 118). The output predictions of the subsystem 133may be used at the motion selector and/or the behavior planner togenerate motion control directives 134 (such as the logical equivalentsof commands to “apply brakes” or “accelerate”) in the depictedembodiment, which may be transmitted to the vehicle motion controlsubsystems 120 to achieve or realize desired movements or trajectories122. It is noted that in one embodiment, at least a portion of one ormore models used for state prediction may be trained at the autonomousvehicle itself, e.g., without requiring resources at remote data centersto be utilized.

Inputs may be collected at various sampling frequencies from individualsensors of the vehicle's sensor collection 112 in different embodimentsvia an intermediary perception subsystem 113 by the behavior planner117, the motion selector 118 and/or the subsystem 133. The perceptionsubsystem may generate higher-level objects or abstractions derived fromthe raw sensor data in various embodiments, which may be moreappropriate for analysis by the decision components than the raw sensordata itself. In one embodiment, an intermediary perception subsystem 113may not be required. Different sensors may be able to update theiroutput at different maximum rates in some embodiments, and as a resultthe rate at which the output derived from the sensors is obtained at thevarious decision making components may also vary from one sensor toanother. A wide variety of sensors may be included in collection 112 inthe depicted embodiment, including externally-oriented cameras,occupant-oriented sensors (which may, for example, include cameraspointed primarily towards occupants' faces, or physiological signaldetectors such as heart rate detectors and the like, and may be able toprovide evidence of the comfort level or stress level of the occupants),Global Positioning System (GPS) devices, radar devices, LIDAR (lightdetection and ranging) devices and so on. In addition to conventionalvideo and/or still cameras, in some embodiment near-infrared camerasand/or depth cameras may be used. In one embodiment, one or more of thecommunication devices 114 may also play the role of a sensor—e.g.,signals regarding the state and/or plans of other autonomous ornon-autonomous vehicles in the vicinity may be collected via anyappropriate communication protocol.

In one approach used for managing the movements of vehicle 110, thebehavior planner 117 may use the state predictions of subsystem 133 togenerate relatively longer-term plans comprising sequences ofconditional actions and states which may be reached as a result of theactions, and provide the alternatives together with associated metadata(e.g., reward or value metrics indicating the “relative goodness” of thealternatives based on currently-known information) to the motionselector 118. The plans may be provided at a slower rate to the motionselector 118 than the rate at which directives 134 are expected to beprovided to the control subsystems 120 in some embodiments (the slowerrate may result, for example, from the amount of computations which haveto be performed to generate the alternatives). As shown in the exampleof FIG. 1 , the ratio of the operating frequencies of the motionselector 118 and the behavior planner 117 may be approximately 10:1 insome embodiments (e.g., if the rate at which updated action sequencesare provided to the motion selector is approximately B Hertz, the rateat which directives are to be provided to the motion control subsystemsmay be approximately 10*B Hertz). Other frequency ratios may be used indifferent embodiments—the ratio shown in FIG. 1 is provided as anexample and is not intended to be limiting. In some embodiments, thebehavior planner 117 may utilize one or more Monte Carlo Tree Search(MCTS) algorithms to generate the plans to be provided to the motionselector. MCTS is an approach for decision making, sometimes used inautomated game player systems, which combines the generality of randomsimulation with the precision of tree search algorithms often employedin machine learning systems.

In some embodiments, at least some of the computations involved ingenerating state predictions at driving time of the autonomous vehicle110 may be performed using remote resources. The communication devices114 (which may comprise hardware and/or software components used forwireless or telephony-based communication channels of any appropriatetypes) may be used to transmit and receive the data corresponding tosuch remote operations, to obtain updates to the models of subsystem 133and so on. The extent to which decision making is performed locallyversus remotely may vary over time in some embodiments—e.g., ifcommunication with a remote facility becomes difficult or unavailable,more decisions (of potentially lower complexity or duration) may begenerated locally temporarily, until communication quality is restored.In one embodiment, regardless of whether decisions are generated atleast in part locally or not, the communication devices 114 may be usedto transmit data about the vehicle's state (and/or local actions takenin various states) to a remote data repository, where the data may laterbe analyzed and utilized to enhance the model(s) 135 or 137 and/or otherdecision components 116.

FIG. 2 illustrates an example decision making scenario for an autonomousvehicle, as well as general problem characteristics associated with suchdecision making scenarios, according to at least some embodiments. Ahigh-level overall route 212 for an autonomous vehicle 250 from anorigin 210 to a destination 215 (labeled “Dest” in FIG. 2 ) may beidentified prior to the start of a journey. The high-level route 212may, for example, be identified using a network-accessible mappingservice, and may comprise information such as a proposed set ofhighways/roads, the exits and entrances to be used for thehighways/roads, and so on. In some cases the high-level route 212 mayalso comprise an estimated or targeted arrival time at the destination215. The route 212 may comprise numerous sub-portions or route segments231, such as segments 231C, 231J and 231V. A given segment 231 mayrepresent, for example, an entrance or turn onto some road or highway,some driving distance on the road or highway, and so on. In someembodiments, one or more tactical maps indicating a set of staticcomponents (including lane segments) of the environment may be obtainedat the vehicle 250 as it traverses a given route segment and analyzedusing graph-based modeling techniques similar to those discussed above.

In the depicted example, route segment 213J comprises the use of anentrance ramp 234 by autonomous vehicle to merge into a multi-lane road233 (the dashed lines indicate lane boundaries). An exit ramp 235 islocated within a short distance of the entrance ramp 234 in the example.The entrance ramp 234, exit ramp 235 and various portions of themulti-lane road may be modeled as lane segments of a tactical map invarious embodiments, as discussed in further detail below. A number ofother vehicles 201 (of which zero or more may be autonomous vehicles),such as vehicles 201A, 201B, 201C, 201D and 201U may be using the road233 at or just before vehicle 250 is to enter the road, and may be closeenough to the entrance ramp to be considered pertinent to the motiondecisions made for vehicle 250. The general directions of motion of thevehicles 201 and 250 are indicated by the arrows—e.g., most of thevehicles shown are assumed to be generally proceeding from left to rightin the view shown in FIG. 2 . The decisions which need to be made withrespect to autonomous vehicle 250 may include, for example, the speed atwhich the road should be entered, how long the vehicle 250 should remainin any given lane immediately after entering the road, whether and towhat extent the vehicle's speed should be changed after the entrance,and so on.

Although experienced human drivers may find these kinds of decisionsfairly straightforward to make, the task of automating such decisions ina timely and safe manner may be complicated. The general problem ofmaking motion-related decisions for the autonomous vehicle may besummarized as follows. Given some level of understanding or knowledgeabout the “world” (the nearby static and dynamic environment of thevehicle 250, as well as the properties of the vehicle's current positionand movements), and given information about the destination 215(including the high-level route 212), the decision making components ofthe vehicle 250 may be required to produce a sequence of trajectoriesfor the vehicle that (a) arrive at the destination safely in a timelyand efficient manner (e.g., meeting some desired fuel or battery usagetargets), (b) are dynamically feasible (e.g., comply with variousapplicable physical laws), (c) follow traffic rules, and (d) provide acomfortable experience for the vehicle's occupants. In order to achievesuch goals, in at least some embodiments neural network-based models ofthe kind discussed in the context of FIG. 1 may be used to forecastfuture states of the world at selected time horizons.

A number of problem characteristics 261 which tend to make the decisionmaking for autonomous vehicles challenging are indicated in FIG. 2 .First, the dynamically changing environment of vehicle 250 may beinherently stochastic rather than deterministic, with noisy rather thanfull and accurate data (such as velocity, position, or heading)available with respect to other vehicles 201 and other relevant objects(such as debris in the road, potholes, signs, etc.). In some cases, oneor more objects 241 may partially block a view of the road. As a result,one or more vehicles such as 201U may be partially or fully unobservedduring the time period in which the merge-related decisions for vehicle250 may have to be made. Furthermore, the intentions or goals of otheragents (e.g., the drivers of vehicles 201, or the decision makingcomponents of those vehicles 201 which happen to be automated vehicles)may be unknown and difficult to predict. For example, even if thecurrent position and velocity of vehicle 201A is known, the agentcontrolling vehicle 201A may suddenly change vehicle 201A's trajectoryto a different lane, as indicated by arrows 211A and 211B, and such atransition may affect the decisions made on behalf of autonomous vehicle250. Second, many of the metrics or observations which may need to beconsidered when making the decisions, such as the velocities andrelative distances between various vehicles, their relationships tonearby elements of the infrastructure or environment, and so on, maytake on continuous rather than discrete values, resulting in atheoretically infinite feasible space of possible states and actions.

The number of sequential actions which may need to be planned may bequite large, resulting in potentially large planning horizons. Inprinciple, to achieve optimal decisions (where the definition ofoptimality may itself be non-trivial), individual actions andcorresponding achievable states may have to be evaluated relative to oneanother with respect to the goals of the journey, and such comparisonsmay become computationally intractable depending on the number ofalternatives being considered at each stage. Finally, because thevehicle 250 is moving, with a high (and therefore potentially dangerous)level of kinetic energy, the decisions may have to be made within tightreal-time constraints, using limited computational resources. Thesecharacteristics, taken together, may make motion planning for autonomousvehicles an extremely difficult proposition, which may require a varietyof algorithms to be utilized in combination as discussed below,including for example algorithms for reasoning about tactical maps.

FIG. 3 illustrates an example overview of the processing of tacticalmaps at an autonomous vehicle, according to at least some embodiments.In the depicted embodiment, raw tactical map information 302 for a givenlocation and/or time with respect to a particular vehicle may compriseinformation about a plurality of static components 304 of the vehicleenvironment. The number and identifiers of relevant static componentssuch as lane segments, intersections, traffic signs and the like may beincluded in the raw data in the depicted embodiment, along with variousattributes or properties of the individual static components such as (inthe case of lane segments) speed limits, recommended speeds, and so on.In addition, the raw tactical map information 302 may compriseinformation about various types of relationships 306 among pairs (and/orother larger groups, such as triples or quadruples) of the staticcomponents. Information about a variety of relationship types, withrespective semantics, may be included in the information 302 for a givenpair or group of static components. The raw tactical map may be madeavailable to (and/or stored at) the computing devices installed at anautonomous vehicle in various formats in different embodiments, forexample in the form of JSON (JavaScript Object Notation) files, XML(Extended Markup Language) files, database entries, custom maprepresentation or markup languages, and so on.

From the raw tactical map, a homogenized graph representation 310 may beproduced in various embodiments. The homogenized graph may comprise aplurality of nodes 312 corresponding to individual static components,and a plurality of edges 314 to represent relationships. A given edge314 may have an edge type or category indicative of the semanticsrepresented by the edge—that is, not all the edges incident on aparticular node may have the same edge type. In at least someembodiments, the individual static components and their properties mayin some embodiments be represented using a matrix in which each rowcorresponds to a static component, and each column corresponds to arespective property. The edges may be represented in some embodiments byadjacency matrices. The graph may be considered homogenized in that allthe nodes may be represented as having the same number and same types ofedges in the depicted embodiment. Thus, for example, if there are Nnodes and E possible edge types to consider in one embodiment, Eadjacency matrices A₁, A₂, . . . , A_(E) may be generated, with eachadjacency matrix comprises N rows and N columns to represent possiblerelationships between each possible pair of nodes. Even if the rawtactical map does not indicate the existence of a relationship of sometype t between nodes i and j, the adjacency matrix A_(t) generated forthe edge type t would still contain an [i, j] entry. The value of suchan [i, j] entry may be set to zero (or to some selected numericalvalue), for example, in some embodiments to indicate that informationregarding the corresponding relationship was absent in the tactical map.

In the depicted embodiment, a deep neural network (DNN) based model 350may be trained to reason over graph representations of tactical mapssimilar to homogenized graph representation 310. Any of a variety ofneural network architectures may be employed singly or in combination indifferent embodiments, such as graph convolutional neural networks(GCNNs), relational networks, non-convolutional graph neural networks,and so on. The DNN-based model(s) 350 may have been trained to produceany of several different types of output in various embodiments, such asresponses to queries pertaining to a vehicle and a set of lane segments,operations a vehicle can legally perform with respect to some set ofstatic components of the environment, etc.

The reasoning results pertaining to the static components of theenvironment, obtained by processing the homogenized graph representationat the DNN-based model(s) 350, may be combined with results obtainedfrom other machine learning models 360 trained for reasoning aboutdynamic or moving entities/components in the depicted embodiment. Any ofa number of different combining algorithms or models 365 may be employedin various embodiments. In one embodiment, different encoding modulesmay be used for the dynamic objects than is used for the static objects.The outputs of such encoding modules may be fed into a large fullyconnected neural network layer which outputs a final decision. In otherapproach employed in some embodiments, the entire environment or “scene”of a vehicle may be treated as a graph that consists of heterogeneoustypes of objects. In such a graph, in addition to the nodes representingstatic components such as lane segments, other nodes may be used torepresent dynamic components such as other vehicles, pedestrians etc.Each node type may be represented by its own feature vector, andadditional edge types may be used to capture relationships between nodesof different types (for example, an edge type that associates a vehicleto a particular lane segment may be introduced, representing arelationship between a dynamic component of the environment and a staticcomponent). The entire environment or scene may then be analyzed using aDNN model 350 in various embodiments. Since different node types mayhave different feature vectors with potentially different lengths, somepreprocessing may be needed to create a feature matrix for the entiregraph in some embodiments. In one embodiment, such pre-processing maycomprise constructing individual feature matrices for each node type,and then creating an aggregate feature matrix by making a block diagonalmatrix of the node type feature matrices. In an equivalent approach usedin some embodiments, every feature may be assigned to every node, andthe ones that are not applicable for a particular node may be set tozero. In at least some embodiments, a set of state predictions,recommended actions or policies 370 may be generated based at leastpartly on the analysis of the tactical maps and partly on the analysisof dynamic entities. Recommended actions may result in, for example,transmission of motion-control directives to vehicle motion controlsubsystems as discussed earlier in the context of FIG. 1 in variousembodiments.

FIG. 4 illustrates example categories of static components of theenvironment which may be included in a tactical map, as well as examplesof properties and relationships of one of the categories, according toat least some embodiments. As shown, categories of static components 402whose analysis may play a role in the planning and control of vehiclemovements may include, among others, lane segments 404 (e.g., portionsof roads or highways), intersections 406, traffic signs/signals 408,pedestrian walkways 410 and so on. In much of the subsequentdescription, lane segments may be used as the primary example of staticcomponents with respect to which various concepts are explained; it isnoted, however, that similar techniques to those discussed withreference to lane segments may be implemented with reference to othertypes of static components in various embodiments. In one embodiment,for example, a combined homogenized graph representation whose nodesrepresent instances of any of several different static componentcategories may be generated (in a manner similar to that discussed inthe context of FIG. 3 for combining dynamic and static components in asingle graph)—e.g., some nodes may represent lane segments, others mayrepresent intersections, and so on. In other embodiments, separatehomogenized graph representations may be created for each type of staticcomponent and analyzed using a respective neural network-based model,with the results of the different models being combined (e.g., using aset of fully connected aggregating neural network nodes which consumethe respective outputs from each of the component-type-specific neuralnetworks).

In various embodiments, lane segment properties 420 for which respectivevalues may be included in a tactical map for a given lane segment mayinclude, for example, a speed limit 422, a proceed-with-cautionindicator 424 (indicating that a vehicle is expected to exercise extracaution while traversing the lane segment), a must-stop indicator 426(indicating that a vehicle must stop when using the lane segment), akeep-clear indicator 428 (indicating that a vehicle must attempt to keepthe lane segment clear), a must-not-enter indicator 430 (indicating thatentry into the lane segment is forbidden), a stop-if-able indicator 432(indicating that a vehicle should stop if possible prior to entering thelane segment), a bi-directional indicator 434 (indicating whether thelane permits bi-directional flow of traffic), one or more lane segmentgeometry indicators 436 (e.g., left, center and/or right poly lines), alane type 438 (e.g., whether use of the lane segment isrestricted/permitted for vehicles, bicycles, etc.), a recommended speed440, and so on. Not all the properties 420 indicated in FIG. 4 may beused in some embodiments. Values for other combinations of propertiesmay be included in a tactical map in some embodiments; the propertiesshown in FIG. 4 are not meant to be restrictive. In the homogenizedgraph representation of a tactical map, in various embodiments a featurevector indicating the values of the different properties for a givenstatic component may be included, as illustrated by the example shownstarting with FIG. 6 below.

In the embodiment depicted in FIG. 4 , relationship categories 450 for agiven pair of lane segments may include, among others, a leftrelationship 452 (indicating whether the second lane segment of the pairis to the left of the first lane segment), a right relationship 454(indicating whether the second lane segment of the pair is to the rightof the first lane segment, a previous relationship 456 (indicatingwhether the second lane segment is a previous lane segment along aparticular path with respect to the first lane segment), and/or a nextrelationship 458 (indicating whether the second lane segment is a nextlane segment along a particular path with respect to the first lanesegment). In some embodiments, other types of relationship categoriesmay also be represented in a tactical map, such as an overlaprelationship 460 indicating whether the second lane segment of a pairoverlaps with the first lane segment, a coincident relationship 462indicating whether the second lane segment is coincident with the firstlane segment, an island relationship indicating 464 whether the secondlane segment represents a traffic island with respect to the first lanesegment, and/or an invalid relationship 466.

In some embodiments, a tactical map may include an indication, withrespect to a given pair of lane segments or a single lane segment, ofpermeability. Lane segment permeability categories 480 may include, forexample, free permeability 482, passing/overtaking 484, illegal 486,barrier 488, turn-across 490, bidirectional turn 492, unknownpermeability 494, and/or invalid permeability 496 in variousembodiments. In at least one embodiment in which permeabilityinformation is included, the set of edge types used in the homogenizedgraph representation of the tactical map may be obtained as across-product of at least some of the relationship categories 450 and atleast some of the permeability categories 480. Other categories ofrelationships and/or permeability categories than those shown in FIG. 4may be used in different embodiments.

FIG. 5 illustrates an example neural network architecture which may beused to process homogenized graph representations of tactical maps,according to at least some embodiments. A convolutional neural networkis illustrated by way of example, comprising an input layer utilizing atleast a portion of the homogenized graph representation of a tacticalmap 502, one or more convolutional network layer groups 510 such as 510Aand 510B, and a fully-connected layer 530 at which results/predictions540 may be generated. Values of various weights, biases and/or otherparameters at the different layers may be adjusted usingback-propagation during the training phase of the model. In someembodiments, other types of deep neural networks capable of reasoningabout graph structures may be used as discussed earlier.

In the depicted embodiment, each convolutional layer group 510 maycomprise a set of nodes 512 (artificial neurons) configured to computeconvolutions, including a non-linear transformation as indicated by theexample propagation function 550 shown for a layer 1. The propagationfunction 550 includes a summation operation to which an element-wisenon-linearity σ 552 is applied. The summation may be performed for all Kedge types being considered (the lowercase letter k is the edge typeindex 554), and may include the multiplication of an adjacency matrixA_(k) 556 with an input matrix or vector H^((l)) 558 for layer land aset of edge-type-specific learned weights W_(k) ^((l)) 560.

In the formulation 550 of the propagation function, for a given edgetype k, a linear combination across feature channels represented by thecolumns of W_(k) ^((l)) is computed, and for each node or row i, A_(k)only allows contributions from nodes that actually share an edge of typek with the node i. The summation is then performed over all the edgetypes in the depicted embodiment. Note that each column of W_(k) ^((l))may correspond to a different convolution. Note also that H⁽⁰⁾ (inputfor the very first layer with index 0) may comprises a feature matrix Xrepresenting the static components for which information was included inthe tactical map in the depicted embodiment.

In some embodiments it may be observed that the eigenvalues of A_(k)might be greater than one, which may cause numerical instabilities. Insuch embodiments, the formulation 550 may be normalized to obtainformulation 550-norm using degree matrices D as follows (D is the degreematrix with

$D_{ii} = {\sum\limits_{j}{\left. A_{ij} \right):}}$ $\begin{matrix}{\sigma\left( {\sum\limits_{k = 0}{D^{{- 1}/2}A_{k}D^{{- 1}/2}H^{(l)}W_{k}^{(l)}}} \right)} & \underline{{{Formulation}550}‐{norm}}\end{matrix}$

In one embodiment, the input feature vector X may be transformed orembedded to increase dimensionality: e.g., the transformation H⁽⁰⁾=ϕ(X;β) may be applied where ϕ embeds each row of X into a high dimensionalspace.

In addition to the computations associated with one or more of theconvolution formulations discussed above, in some embodiments a givenlayer group 510 may comprise pooling-related computations at an optionalpooling layer. As shown in FIG. 5 , in various embodiments, multiplelayer groups 510 may be arranged in sequence, with the output of onelayer group such as 510A being consumed as input by the next layer groupsuch as 510B. In some embodiments, a different arrangement ofconvolutional layers, non-linear transformation layers such asrectifying linear units (ReLUs) and/or pooling layers may be used—e.g.,a sequence of layers similar to the following may be used:Convolution(layer#1)-ReLU(#2)-Convolution(#3)-ReLU(#4)-Pooling(#5)-Convolution(#6)-ReLU(#7)-Convolution(#8)-ReLU(#9)-Pooling(#10)-FullyConnected(#11).

A small-scale example of the types of computations which may beimplemented, starting with a tactical map, to generate a homogenizedgraph representation on which reasoning is performed by a neural networkis provided using FIG. 6 -FIG. 9 . FIG. 6 illustrates a simple exampleof a feature matrix which may be generated to represent nodes of ahomogenized graph corresponding to a tactical map, according to at leastsome embodiments. In the example scenario, information about six lanesegments LS1-LS6, for each of which values of two properties 604(recommended speed RS and proceed-with-caution indicator PWC) are known,may be included in a tactical map. To keep the example simple, a verysmall number of static components of a vehicle's environment in the formof these six lane segments is presented, and the number of properties tobe considered is also very small. In real-world scenarios, a largernumber of static components and properties may be included in a tacticalmap. In some embodiments, as mentioned earlier, one or more types ofstatic components in addition to lane segments (or instead of lanesegments) may be included in at least some tactical maps and thecorresponding graph representations.

In the example scenario shown, the RS or recommended speed values may beexpressed as integers (e.g., indicating miles-per-hour orkilometers-per-hour), while the PWC or proceed-with-caution indicatormay be expressed as a Boolean (e.g., with a “1” value indicating thatextra caution is recommended, and a “0” value indicating that normallevels of caution are sufficient). For each of lane segments LS1, LS2and LS3, as shown by elements 604A-604C respectively, the exampletactical map may indicate that RS=60 and PWC=0. For each of lanesegments LS4, LS5 and LS6, (as shown by elements 604D-604F) RS=40 whilePWC=1. The lane segments LS4-LS6 are more closely associated withentering (via entrance ramp 634) and exiting (via exit ramp 635) theroad, which may lead to the need for greater caution and lowerrecommended driving speeds. The information about the lane segments andtheir properties may be transmitted to a vehicle moving on one of thesix lane segments, or in the vicinity of the six lane segments, invarious embodiments via a network connection, or may be stored inadvance of a journey at computing devices installed in the vehicle.

A feature matrix X indicated by label 640 may be created to representthe nodes of the graph corresponding to the tactical map in the depictedembodiment. Corresponding to each of the six lane segments, a featurevector xi indicating the RS value and the PWC value may be generated,and the feature vectors may be transposed and combined forming featurematrix X as shown. The rows 642 of X represent individual lane segments,while the columns 644 represent individual properties. The third rowfrom the top, x₃ ^(T), represents LS3, for example, as indicated byarrow 655.

FIG. 7 illustrates a set of adjacency matrices which may be generated torepresent edges of the homogenized graph corresponding to the tacticalmap introduced in FIG. 6 , according to at least some embodiments. Tosimplify the example, only four types of relationship semantics amonglane segment pairs, with corresponding edge types 702, are considered:Left, Right, Previous and Next edge types. Recall that the examplecomprises information about six lane segments or nodes. Corresponding toeach of the relationship semantics types, a respective 6×6 adjacencymatrix A1-A4 may be constructed to represent the edges in the depictedembodiment. The relative position of the entry for a given node is basedon the node label—e.g., LS1 is represented by the first row and firstcolumn in the adjacency matrices, LS2 is represented by the second rowand second column, and so on. The matrix A1 is used for the “Left”relationship, A2 for the “Right” relationship, A3 for the “Previous”relationship and A4 for the “Next” relationship. (The exact meanings orsemantics of “Previous” and Next” may be determined by the entity thatgenerates the tactical map—intuitively, for example, a first lanesegment may be considered previous to a second lane segment if a vehiclewhich does not change lanes to the left or right would transition fromthe first lane segment to the second lane segment in some embodiments).In some embodiments, to simplify convolution-related calculations, aself-edge matrix A0 (the identity matrix of the same dimensions as theA1-A4 matrices) may also be defined as indicated in FIG. 7 .

A visual examination of the positional/geometrical arrangement of thesix lane segments as shown in FIG. 7 indicates that the only direct“Left” relationship is between segments LS2 and LS5-LS2 is to the leftof LS5, and conversely, LS5 is to the right of LS2. Accordingly(assuming row and column numbering starts with 1) only the [row=5,column=2] element of A1 is set to 1, with all other elements being setto zero. Similarly, only the [row=2, column=5] element of A2 (for the“Right” relationship) is set to 1, with all other elements being zero.With respect to the “Previous” relationship, A3 has non-zero values onlyin elements [2,1], [3,2], [5,4] and [6,5], indicating that LS1 isprevious to LS2, LS2 is previous to LS3, LS4 is previous to LS5 and LS5is previous to LS6. With respect to the “Next” relationship, A4 hasnon-zero values only in elements [1,2], [2,3], [4,5] and [5,6],indicating that LS2 is next to LS1, LS3 is next to LS2, LS5 is next toLS4 and LS6 is next to LS5. Note that in the homogenized graphrepresentation, there are entries in the adjacency matrices for allpairs of nodes for each of the possible relationship types, even thoughin the tactical map only information about existing or actualrelationships may have been included in the depicted embodiment. Ineffect, the number and types of edges associated with any given node ismade equal to the number and types of edges associated with any othernode of the homogenized graph in such embodiments.

FIG. 8 illustrates an intermediate summation operation associated with apropagation function of a first convolution layer at a neuralnetwork-based machine learning model to which the homogenized graphdiscussed in the context of FIG. 6 and FIG. 7 is provided as input,according to at least some embodiments. Recall that in formulation 550shown in FIG. 5 , the input H^((l)) for l=0 (the initial or firstconvolution layer) may comprise the feature matrix X representing thenodes of the homogenized graph. Using the values for the adjacencymatrices shown in FIG. 7 and the feature matrix shown in FIG. 6 , thesum 802 may be computed as shown. This linear sum of the products of theedge-type-specific adjacency matrices, the feature vector X (receivedfrom the input layer of the network) and a vector w_(k) representinglearned weights associated with respective edge types, summed over theedge types 0 through 4, may in turn be processed using a non-lineartransformation σ in some embodiments to obtain the overall propagationfunction. The linear combination may be computed for every node of thegraph in the depicted embodiment, which may correspond to a convolutionstride of one.

It is noted that one or more weights may be shared in different rows ofthe matrix 806 obtained from the homogenized graph representation of thetactical map in the depicted embodiment—e.g., w₄ occurs in rows 1, 2, 4and 5, w₃ occurs in rows 3, 5 and 6, and so on. This weight-sharing may,for example, result in more efficient learning than if all the weightsfor all of the nodes were different. In some embodiments, pooling mayoptionally be implemented at the neural network by combining lanesegments using a set of defined rules. Note that as presented in FIG. 8, only nodes that are one “hop” away (share an immediate edge) arepresent in the convolution. To increase the window size of theconvolution, in some embodiments pairs of adjacency matrices may bemultiplied with one another (e.g., a new adjacency matrix A5 may begenerated by multiplying A4 and A1).

In at least some embodiments, as discussed earlier, multiple layers orcompositions (such as composition 804) may be applied, e.g., usingrespective layer groups of neural network nodes as discussed earlier.The output of one convolution layer (e.g., level-0) may be consumed asthe input (H) of the next layer (e.g., level-1) in such embodiments. Thenumber of convolution layers used may, for example, depend on the typesand granularities of the queries which are to be answered (which in turncorrespond to the kinds of reasoning to be performed) using the neuralnetwork-based model in different embodiments. In FIG. 8 , the notationw_(k) for k=0, . . . , K (in this example K=4) of sum 802 corresponds toone convolution, while the notation W_(k) for k=0, . . . , K of thecomposition 804 corresponds to multiple convolutions.

FIG. 9 illustrates example queries to which responses may be generatedusing a neural network-based machine learning model similar to thatdiscussed in the content of FIG. 8 , according to at least someembodiments. In various embodiments, the neural network-based model maybe trained to reason about graph data structures, and provide one ormore different types of results of such reasoning. In some embodiments,such results may comprise respective responses to one or more types ofqueries pertaining to the vehicle and the environment componentsrepresented in the tactical map and homogenized graph. In at least oneembodiments, respective scores associated with several differentpotential specific responses to a given query may be generated. Thespecific responses with the highest scores may be used, e.g., incombination with results of other analyses of moving objects and thelike, to determine motion control directives to be used to achievedesired vehicle trajectories.

In the depicted embodiment, three example queries 902 (query 902A, 902Band 902C) are shown, for which responses may be generated using atrained deep neural network model such as a graph convolutional neuralnetwork of the kind discussed earlier. Some queries, such as 902A, mayrequest geometrical or topological information about static componentsincluding lane segments and the like: e.g., the total arc length alongsome combination of adjacent lane segments (LS1, LS2, LS3) may berequested in query 902A. Other queries may request information aboutfeasible or legal movements of the vehicle with respect to some set ofstatic components—e.g., query 902B may pertain to the legality ofexiting from the road/highway via a particular lane segment (LS3). Somequeries may require combining additional information, e.g., results ofanalysis of dynamic/moving objects, with the reasoning regarding staticcomponents of the tactical map—e.g., the query 902C regarding thecurrent safest path from one lane segment to another. In at least someembodiments, outputs of a neural network based model trained to reasonabout graph representations of tactical maps may comprise other types ofobjects or data. For example, in one embodiment, a reproduction of thetactical map or corresponding homogenized graph may be generated as partof the output using learned weights in internal or hidden layers of themodel, with the accuracy of the reproduction indicating how well themodel has learned or reasoned about the input.

FIG. 10 illustrates an overview of example stages of developing,deploying and using machine learning models for autonomous vehicles,according to at least some embodiments. As shown, as part of acontinuous or ongoing data collection procedure 1001, data about drivingenvironments may be collected from a variety of vehicle-based sensors innumerous geographical regions such as R1 and R2. The set of sensors at agiven vehicle may comprise, among others, externally-oriented cameras,occupant-oriented sensors (which may, for example, include cameraspointed primarily towards occupants' faces, or physiological signaldetectors such as heart rate detectors and the like, and may be able toprovide evidence of the comfort level or stress level of the occupants),Global Positioning System (GPS) devices, radar devices, LIDAR (lightdetection and ranging) devices, sensors associated with vehiclemotion-control subsystems such as brakes, accelerator pedals, steeringwheels, and so on. In addition to conventional video and/or stillcameras, in some embodiment near-infrared cameras and/or depth camerasmay be used. The raw sensor data may be transformed into higher levelabstractions or objects prior to further analysis in at least someembodiments, e.g., using one or more perception subsystems. In additionto the vehicle sensors, data about driving environments may also becollected from extra-vehicular data sources 1005 in at least someembodiments. Such data sources may include, for example, databases oftraffic laws, accident reports, mapping services and the like. In orderto train neural network models for homogenized graph representations ofstatic components of vehicle environments, a number of tactical maps maybe obtained from sources 1033 in at least some embodiments.

The data may be aggregated at one or more primary model training datacenters 1020 in the depicted embodiment. The data centers may comprisenumerous computing platforms, storage platforms and the like, from whichsome number of training platforms 1022 may be selected to train andevaluate neural network-based models 650 using any of a variety ofmachine learning algorithms of a library 1024. Trained models 1050,which may for example the types of models discussed earlier, may betransmitted to autonomous vehicles 1072 (e.g., AV 1072A-1072C) of fleets1070 in the depicted embodiment. The trained models may be executedusing local computing resources at the autonomous vehicle, the currentset of relevant tactical maps 1035 and data collected by local sensorsof the autonomous vehicles, e.g., to predict vehicle environment states,generate motion control directives to achieve vehicle trajectories whichmeet safety, efficiency and other desired criteria, and so on. At leasta subset of the decisions made at the vehicle, as well as the localsensor data collected, may be transmitted back to the data centers aspart of the ongoing data collection approach, and uses to improve andupdate the models in various embodiments. In some embodiments, updatedversions of the models may be transmitted to the autonomous vehiclefleet from the data centers periodically, e.g., as improvements in themodel accuracy and/or efficiency are achieved. In at least oneembodiment, respective versions of models may be generated for severalsub-domains—e.g., one model may be generated for large trucks, anotherfor passenger cars, and so on, and the appropriate models may betransmitted to autonomous vehicles of different types. Similarly, insome embodiments, geography-specific models may be generated andtransmitted to autonomous vehicles for execution in the correspondingregions or countries.

FIG. 11 is a flow diagram illustrating aspects of operations which maybe performed to analyze graph representations of tactical maps atautonomous vehicles using neural network-based machine learning models,according to at least some embodiments. As shown in element 1102, atactical map comprising information about static components of theoperating environment of a vehicle may be obtained at one or morecomputing devices. The tactical map may, for example, compriseinformation about objects/entities such as road/highway lane segments,intersections, traffic signs and the like, attributes of individualobjects, and relationships (e.g., geometric or topographicalrelationships) among pairs or groups of the objects. Any of a number offormats or languages may be used for the raw tactical maps— e.g.,variants of JSON, XML, or proprietary map markup languages may be used.

A homogenized graph representation of the tactical map, in which thestatic components are represented by nodes, and edges with respectiveedge types indicate different semantic relationships among pairs ofstatic components may be generated (element 1104). The graph may be saidto be homogenized in that each node may be represented as having thesame number and type of edges, arranged in the same order in someembodiments. In at least one embodiment, at least some of thecomputations involved in generating the homogenized graph representationmay be performed using resources outside the vehicle—e.g., a homogenizedgraph generator intermediary device at a data center or a tactical mapsource may convert the raw tactical maps into the homogenized graphs andthen transmit them to the vehicle.

The homogenized graph representation may be provided as input to aneural network model (such as a convolutional graph neural network, amore general graph neural network, a relational network or the like asdiscussed earlier) which has been trained to reason on graph datastructures (element 1107). Results generated by the neural network model(such as responses to queries of the kinds discussed above) may be used,in combination with results of other models trained foranalysis/prediction regarding dynamic entities, sensor data analysis andthe like, to generate motion control directives in at least someembodiments (element 1110). The directives may be transmitted to vehiclecontrol subsystems (e.g., subsystems for braking/steering/acceleratingthe vehicle) to cause the vehicle to move along a particular path ortrajectory in various embodiments (element 1113).

It is noted that in various embodiments, at least some operations otherthan those illustrated in the flow diagram of FIG. 11 may be used toimplement the techniques for reasoning regarding graph representationsof tactical maps described above. Some of the operations shown may notbe implemented in some embodiments or may be implemented in a differentorder, or in parallel rather than sequentially.

In some embodiments, computations of the modeling and analysiscomponents associated with an autonomous vehicle may be performed in adistributed manner, e.g., using local resources located in the vehicleitself as well as remote resources. A remote data center may comprise,for example, one or more pools of compute resources, such as virtual orphysical servers. In some embodiments, at least some resource pools orindividual servers of a data center may be optimized for performingmachine learning operations—e.g., some servers may use graphicalprocessing units (GPUs) more suitable for neural network algorithms thanconventional CPUs. The resources of the data center may be used to trainand run at least some of the machine learning models (including forexample the kinds of deep neural networks discussed above) used by thelocal decision making components of the autonomous vehicles; thus, thedecision making components in effect may comprise some on-boardcomponents local to the vehicles and some remote components in someembodiments. The data center may also store one or more databases (e.g.,comprising tactical maps, local regulations, and the like for some areaor region) in some embodiments.

A given autonomous vehicle may comprise a respective set of on-boardcomputing resources, as well as a respective dynamically-updated localcache in some embodiments. Depending for example on the current locationof a vehicle and/or the identities of the current occupants, the cachesmay be updated periodically from a data center's databases. For example,city-specific or neighborhood-specific traffic rules may be loaded intothe caches as needed, occupant preferences (such as the relative weightsto be assigned to different factors when selecting actions to be taken)may be updated when occupants change, and so on. It is noted that notall the autonomous vehicles may incorporate equivalent local computingresources—e.g., some vehicle models may contain more powerful processingdevices and/or larger caches than others.

Based at least in part on the quality of the network connectivitybetween the autonomous vehicles and the data center, different relativeamounts of motion decision-related processing may be performed locallyusing on-board resources versus remotely in various embodiments. Ingeneral, the autonomous vehicles may attempt to establish and maintainhigh-bandwidth and low-latency communication channels with the datacenter. If the quality of the connectivity is high, data may betransmissible at a high rate and with high fidelity to/from the datacenter, and as a result a larger fraction of the computations needed forvehicle motion-related decisions, may be performed at the data center.In contrast, if the connectivity is of lower quality, a larger fractionof the decision-making logic may have to be implemented using on-boardresources. In at least one embodiment, the tasks of collecting andprocessing sensor data, and the motion selector's operations, may beperformed using local computing resources, while it may be possible(although not required) to perform at least a fraction of higher-levelstate prediction functions and/or behavior planner functions at the datacenter. In various embodiments, because of the possibility of poorconnectivity with the remote data center, the decision-making componentsof the vehicle may be designed such that they are capable of planningthe vehicle's movements appropriately and safely even if communicationswith the data center are not feasible for extended periods of time.

In at least some embodiments, a server that implements a portion or allof one or more of the techniques for analyzing and reasoning abouttactical maps described herein, including the techniques to collect,obtain or process sensor signals of various types, to generatehomogenized graph representations of tactical maps, train and executemachine learning algorithms including neural network algorithms and thelike, may include a general-purpose computer system that includes or isconfigured to access one or more computer-accessible media. FIG. 12illustrates such a general-purpose computing device 9000. In theillustrated embodiment, computing device 9000 includes one or moreprocessors 9010 coupled to a main memory 9020 (which may comprise bothnon-volatile and volatile memory modules, and may also be referred to assystem memory) via an input/output (I/O) interface 9030. Computingdevice 9000 further includes a network interface 9040 coupled to I/Ointerface 9030, as well as additional I/O devices 9035 which may includesensors of various types.

In various embodiments, computing device 9000 may be a uniprocessorsystem including one processor 9010, or a multiprocessor systemincluding several processors 9010 (e.g., two, four, eight, or anothersuitable number). Processors 9010 may be any suitable processors capableof executing instructions. For example, in various embodiments,processors 9010 may be general-purpose or embedded processorsimplementing any of a variety of instruction set architectures (ISAs),such as the x86, PowerPC, SPARC, or MIPS ISAs, or any other suitableISA. In multiprocessor systems, each of processors 9010 may commonly,but not necessarily, implement the same ISA. In some implementations,graphics processing units (GPUs) may be used instead of, or in additionto, conventional processors.

Memory 9020 may be configured to store instructions and data accessibleby processor(s) 9010. In at least some embodiments, the memory 9020 maycomprise both volatile and non-volatile portions; in other embodiments,only volatile memory may be used. In various embodiments, the volatileportion of system memory 9020 may be implemented using any suitablememory technology, such as static random access memory (SRAM),synchronous dynamic RAM or any other type of memory. For thenon-volatile portion of system memory (which may comprise one or moreNVDIMMs, for example), in some embodiments flash-based memory devices,including NAND-flash devices, may be used. In at least some embodiments,the non-volatile portion of the system memory may include a powersource, such as a supercapacitor or other power storage device (e.g., abattery). In various embodiments, memristor based resistive randomaccess memory (ReRAM), three-dimensional NAND technologies,Ferroelectric RAM, magnetoresistive RAM (MRAM), or any of various typesof phase change memory (PCM) may be used at least for the non-volatileportion of system memory. In the illustrated embodiment, executableprogram instructions 9025 and data 1926 implementing one or more desiredfunctions, such as those methods, techniques, and data described above,are shown stored within main memory 9020.

In one embodiment, I/O interface 9030 may be configured to coordinateI/O traffic between processor 9010, main memory 9020, and variousperipheral devices, including network interface 9040 or other peripheralinterfaces such as various types of persistent and/or volatile storagedevices, sensor devices, etc. In some embodiments, I/O interface 9030may perform any necessary protocol, timing or other data transformationsto convert data signals from one component (e.g., main memory 9020) intoa format suitable for use by another component (e.g., processor 9010).In some embodiments, I/O interface 9030 may include support for devicesattached through various types of peripheral buses, such as a variant ofthe Peripheral Component Interconnect (PCI) bus standard or theUniversal Serial Bus (USB) standard, for example. In some embodiments,the function of I/O interface 9030 may be split into two or moreseparate components, such as a north bridge and a south bridge, forexample. Also, in some embodiments some or all of the functionality ofI/O interface 9030, such as an interface to memory 9020, may beincorporated directly into processor 9010.

Network interface 9040 may be configured to allow data to be exchangedbetween computing device 9000 and other devices 9060 attached to anetwork or networks 9050, such as other computer systems or devices asillustrated in FIG. 1 through FIG. 11 , for example. In variousembodiments, network interface 9040 may support communication via anysuitable wired or wireless general data networks, such as types ofEthernet network, for example. Additionally, network interface 9040 maysupport communication via telecommunications/telephony networks such asanalog voice networks or digital fiber communications networks, viastorage area networks such as Fibre Channel SANs, or via any othersuitable type of network and/or protocol.

In some embodiments, main memory 9020 may be one embodiment of acomputer-accessible medium configured to store program instructions anddata as described above for FIG. 1 through FIG. 11 for implementingembodiments of the corresponding methods and apparatus. However, inother embodiments, program instructions and/or data may be received,sent or stored upon different types of computer-accessible media.Generally speaking, a computer-accessible medium may includenon-transitory storage media or memory media such as magnetic or opticalmedia, e.g., disk or DVD/CD coupled to computing device 9000 via I/Ointerface 9030. A non-transitory computer-accessible storage medium mayalso include any volatile or non-volatile media such as RAM (e.g. SDRAM,DDR SDRAM, RDRAM, SRAM, etc.), ROM, etc., that may be included in someembodiments of computing device 9000 as main memory 9020 or another typeof memory. Further, a computer-accessible medium may includetransmission media or signals such as electrical, electromagnetic, ordigital signals, conveyed via a communication medium such as a networkand/or a wireless link, such as may be implemented via network interface9040. Portions or all of multiple computing devices such as thatillustrated in FIG. 12 , and/or multiple storage media may be used toimplement the described functionality in various embodiments; forexample, software components running on a variety of different devicesand servers using one or more processors may collaborate to provide thefunctionality. In some embodiments, portions of the describedfunctionality may be implemented using storage devices, network devices,or special-purpose computer systems, in addition to or instead of beingimplemented using general-purpose computer systems. The term “computingdevice”, as used herein, refers to at least all these types of devices,and is not limited to these types of devices.

CONCLUSION

Various embodiments may further include receiving, sending or storinginstructions and/or data implemented in accordance with the foregoingdescription upon a computer-accessible medium. Generally speaking, acomputer-accessible medium may include storage media or memory mediasuch as magnetic or optical media, e.g., disk or DVD/CD-ROM, volatile ornon-volatile media such as RAM (e.g. SDRAM, DDR, RDRAM, SRAM, etc.),ROM, etc., as well as transmission media or signals such as electrical,electromagnetic, or digital signals, conveyed via a communication mediumsuch as network and/or a wireless link.

The various methods as illustrated in the figures and described hereinrepresent exemplary embodiments of methods. The methods may beimplemented in software, hardware, or a combination thereof. The orderof method may be changed, and various elements may be added, reordered,combined, omitted, modified, etc.

Various modifications and changes may be made as would be obvious to aperson skilled in the art having the benefit of this disclosure. It isintended to embrace all such modifications and changes and, accordingly,the above description to be regarded in an illustrative rather than arestrictive sense.

1.-20. (canceled)
 21. A method, comprising: generating a graphrepresentation of one or more components of an environment of a vehicle;obtaining, from one or more machine learning models, a response to aquery pertaining to safety or traffic rules of a particular path of thevehicle, wherein input to the one or more machine learning modelsincludes the graph representation; and causing, based at least in parton the response to the query, the vehicle to move along the particularpath.
 22. The method as recited in claim 21, wherein the one or morecomponents of the environment include at least one static component. 23.The method as recited in claim 22, wherein the static componentcomprises one of: (a) a lane segment of a road, (b) an intersection, (c)a traffic sign, (d) a traffic signal or (e) a pedestrian walkway. 24.The method as recited in claim 21, wherein the graph representationcomprises an edge representing one of: (a) a geometric constraintassociated with at least a pair of static components of the environment,(b) a topological constraint associated with at least a pair of staticcomponents of the environment or (c) one or more attributes associatedwith at least a pair of static components of the environment.
 25. Themethod as recited in claim 21, wherein the one or more machine learningmodels include a neural network-based machine learning model.
 26. Themethod as recited in claim 21, further comprising: analyzing, using theone or more machine-learning models, a representation of one or moremoving objects in the environment of the vehicle, wherein causing thevehicle to move along the particular path is based at least in part on aresult of said analyzing.
 27. The method as recited in claim 21, whereinthe graph comprises (a) a first node representing a first component ofthe environment, (b) a second node representing a second component ofthe environment, (c) a first edge belonging to a first edge typeindicative of a relationship of a first semantic category between thefirst node and the second node, and (d) a second edge belonging to asecond edge type indicative of a relationship of a second semanticcategory between the first node and the second node.
 28. A system,comprising: one or more processors; and one or more memories; whereinthe one or more memories store program instructions that when executedon or across the one or more processors perform a method comprising:generating a graph representation of one or more components of anenvironment of a vehicle; obtaining from one or more machine learningmodels, a response to a query pertaining to safety or traffic rules of aparticular path of the vehicle, wherein input to the one or more machinelearning models includes the graph representation; and causing, based atleast in part on the response to the query, the vehicle to move alongthe particular path.
 29. The system as recited in claim 28, wherein theone or more components of the environment include at least one staticcomponent.
 30. The system as recited in claim 29, wherein the staticcomponent comprises one of: (a) a lane segment of a road, (b) anintersection, (c) a traffic sign, (d) a traffic signal or (e) apedestrian walkway.
 31. The system as recited in claim 28, wherein thegraph representation comprises an edge representing one of: (a) ageometric constraint associated with at least a pair of staticcomponents of the environment, (b) a topological constraint associatedwith at least a pair of static components of the environment or (c) oneor more attributes associated with at least a pair of static componentsof the environment.
 32. The system as recited in claim 28, wherein theone or more machine learning models include a neural network-basedmachine learning model.
 33. The system as recited in claim 28, whereinthe method further comprises: analyzing, using the one or moremachine-learning models, a representation of one or more moving objectsin the environment of the vehicle, wherein causing the vehicle to movealong the particular path is based at least in part on a result of saidanalyzing.
 34. The system as recited in claim 28, wherein the graphcomprises (a) a first node representing a first component of theenvironment, (b) a second node representing a second component of theenvironment, (c) a first edge belonging to a first edge type indicativeof a relationship of a first semantic category between the first nodeand the second node, and (d) a second edge belonging to a second edgetype indicative of a relationship of a second semantic category betweenthe first node and the second node.
 35. One or more non-transitorycomputer-accessible storage media storing program instructions that whenexecuted on or across one or more processors cause the one or moreprocessors to perform a method comprising: generating a graphrepresentation of one or more components of an environment of thevehicle; obtaining from one or more machine learning models, a responseto a query pertaining to safety or traffic rules of a particular path ofthe vehicle, wherein input to the one or more machine learning modelsincludes the graph representation; and instructing, based at least inpart on the response to the query, the vehicle to move along theparticular path.
 36. The one or more non-transitory computer-accessiblestorage media as recited in claim 35, wherein the one or more componentsof the environment include at least one static component.
 37. The one ormore non-transitory computer-accessible storage media as recited inclaim 36, wherein the static component comprises one of: (a) a lanesegment of a road, (b) an intersection, (c) a traffic sign, (d) atraffic signal or (e) a pedestrian walkway.
 38. The one or morenon-transitory computer-accessible storage media as recited in claim 35,wherein the graph representation comprises an edge representing one of:(a) a geometric constraint associated with at least a pair of staticcomponents of the environment, (b) a topological constraint associatedwith at least a pair of static components of the environment or (c) oneor more attributes associated with at least a pair of static componentsof the environment.
 39. The one or more non-transitorycomputer-accessible storage media as recited in claim 35, wherein theone or more machine learning models include a neural network-basedmachine learning model.
 40. The one or more non-transitorycomputer-accessible storage media as recited in claim 35, wherein themethod further comprises: analyzing, using the one or moremachine-learning models, a representation of one or more moving objectsin the environment of the vehicle, wherein causing the vehicle to movealong the particular path is based at least in part on a result of saidanalyzing.